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Abstract 

Along with advancing mobile technologies and proliferating mobile devices and applications, 
mobile learning research has gained great momentum in recent years. While there have been 
review articles summarizing past research, studies identifying mobile learning research priorities 
based on experts’ latest insights have been lacking. This study employed the Delphi method to 
obtain a consensus from experts about areas that are most in need of research in mobile learning. 
An international expert panel participated in a three-round Delphi process involving two cycles 
of online questionnaires and feedback reports. Participants responded to the question, “What 
should be the research priorities for the field of mobile learning over the next 5 years?” Ten 
research categories were identified and ranked in order of priority: 1) teaching and learning 
strategies; 2) affordances; 3) theory; 4) settings of learning; 5) evaluation/assessment; 6) 
learners; 7) mobile technologies and interface design; 8) context awareness and augmented 
reality; 9) infrastructure and management; and 10) country and digital divide. This study also 
reported expert-generated research statements for each research category and the importance of 
these research statements rated by the experts themselves. Selected research papers were 
summarized to help contextualize the discussions of research categories and statements. 

Resume 

Avec l’avancement des technologies mobiles et la proliferation des appareils mobiles et des 
applications, la recherche consacree a I’apprentissage mobile a recemment pris de Tampleur. Si 
des articles ont resume les recherches anterieures, les etudes s’appuyant sur les dernieres 
connaissances d’experts pour identifier les priorites de recherche sur I'apprentissage mobile font 
defaut. La presente etude a utilise la methode de Delphes pour obtenir un consensus des experts 
sur les domaines necessitant le plus des recherches sur I’apprentissage mobile. Un groupe 
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international d’experts a participe a un processus de Delphes structure en trois rondes impliquant 
deux series de questionnaires en ligne et des rapports de retroaction. Les participants ont repondu 
a la question : "Quelles devraient etre les priorites de recherche dans le domaine de 
l'apprentissage mobile pour les cinq prochaines annees ?" Dix categories de recherche ont ete 
identifiees et classees par ordre de priorite : 1 ) strategies d'enseignement et d’apprentissage ; 2 ) 
affordances ; 3 ) theorie ; 4 ) parametres d’apprentissage ; 5 ) evaluation ; 6 ) apprenants ; 7 ) 
technologies mobiles et conception de l’interface ; 8 ) perception du contexte et realite augmentee 
; 9 ) infrastructure et gestion ; et 10 ) pays et fosse numerique. Cette etude a egalement repris les 
declarations de recherche enoncees pour chaque categorie par les experts ainsi que le classement 
par ordre d’importance des declarations de recherche selon l’avis de ces experts. Quelques 
articles choisis ont ete resumes pour faciliter la contextualisation des discussions portant sur les 
categories de recherche et sur les declarations. 


Introduction 

The rapid advancement in mobile technologies in recent years has made innovation in 
mobile learning possible due to varied affordances. O’Malley et al. (2005) defined mobile 
learning as “any sort of learning that happens when the learner is not at a fixed, predetermined 
location, or learning that happens when the learner takes advantage of the learning opportunities 
offered by mobile technologies.” Both are learner-centric, but the first half of the definition 
points out the mobility and ubiquity aspects of mobile learning, while the second half relates to 
the concept of affordances. Affordances refer to the tasks or activities made possible by the 
features or functions of technologies (Hsu, Ching, & Grabowski, 2014). The major affordances 
of mobile technologies for learning include the following aspects: 1) high device-portability that 
enables easy access to mobile devices and user mobility (Brown, 2009); 2) relatively strong 
computing power that gives learners the ability to achieve and complete tasks on small devices 
with a capability equivalent to larger and less portable devices (Lai & Wu, 2006); 3) always-on 
and stable Internet connectivity with high bandwidth which allows for instant access to large 
amounts of infonnation and real-time communication regardless of location (Johnson, Smith, 
Willis, Levine, & Haywood, 2011). These affordances of mobile technologies make it possible 
for learning to occur in the real-world contexts relevant to learners. Learners can take the mobile 
devices anywhere they want to execute tasks or continue their learning processes beyond 
classrooms. Learning can be ubiquitous and seamless because of the portability and strong 
computing power of mobile technologies, (Liu, Tan, & Chu, 2009). In addition, the constant 
Internet connectivity of mobile devices can help promote social learning through communication 
and collaboration among learners (Zurita & Nussbaum, 2004a). Through sharing knowledge and 
experiences, learners can develop expertise related to their field or their interest (Lave & 

Wenger, 1991). Mobile devices afford rich and varied communication and collaboration 
possibilities (Motiwalla, 2007) that are critical to collaborative knowledge construction. 

Parallel to the year-to-year increasing sales number of mobile devices (Gartner, 2011), 
advancement in mobile technologies and features (Wu, Wu, Cheng, Kao, Lin, & Huang, 2012), 
and availability of various mobile applications (Hsu, Rice, & Dawley, 2012), there has been 
growing interest in the educational use of mobile devices/technologies (Rushby, 2012). 

However, implementing mobile learning, especially regarding the evaluation portion of it, can 
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have challenges due to the ubiquitous nature of mobile learning. Vavoula and Sharpies (2009) 
suggested six challenges in evaluating mobile learning, including capturing and analyzing 
learning in context and across contexts, measuring mobile learning processes and outcomes, 
respecting learner/participant privacy, assessing mobile technology utility and usability, 
considering the wider organizational and socio-cultural context of learning, and assessing 
in/formality. Kukulska-Hume (2013) also pointed out issues such as privacy, security and trust, 
and interdependencies with the commercial world, which related to mobile apps sending user 
information back to app developers and advertisement/analytics companies. Other issues and 
challenges include classroom management (Kukulska-Hume, 2013), which is complicated by the 
portability of mobile devices, and further complicated by social media (e.g., YouTube) and 
social networking applications (e.g., Facebook) available on mobile devices that could distract 
students during their learning. Also, with the combination of mobile devices and social media 
applications, there are ethical issues where users/learners can distribute recorded image, audio, 
and video to social networking sites more easily than ever. These challenges and issues all need 
to be addressed with appropriate corresponding learning design and pedagogy. 

The productivity in mobile learning research (Kukulska-Hulme & Traxler, 2007) that 
examines learning potential and related critical issues leads to an emerging body of literature. 
This body of research has started to identify trends of mobile learning research using methods 
such as content analysis through categorization and classification (e.g., Hwang & Tsai, 2011; Wu 
et ah, 2012) and text mining (e.g., Hung & Zhang, 2012). Those literature reviews examined 
published articles in journals and proceedings to identify research themes and 
increasing/decreasing trends. Hwang and Tsai (2011) reviewed 154 articles published between 
2001 and 2010 from 6 journals included in the Social Science Citation Index (SSCI). They found 
that most studies focused on motivation, perceptions and attitudes of students toward mobile and 
ubiquitous learning. They also found that mobile learning research conducted between 2006 and 
2010 has increased significantly in numbers of publications across learning domains such as 
engineering (including computers), arts and language, science, and social sciences, compared 
with research conducted between 2001 and 2005. Hung and Zhang (2012) examined 119 
refereed journal articles and proceedings (published between 2003 and 2008) retrieved from 
Science Citation Index (SCI) and SSCI databases, and found that research during that period 
focused on examining effectiveness of mobile learning, adaptive or intelligent tutoring systems, 
and personalized mobile learning systems. Wu et al. (2012) analyzed 164 articles published 
between 2003 and 2010, retrieved from multiple databases and refereed journals. Similar to the 
findings of Hung and Zhang, Wu et al. found that most mobile learning research focused on 
effectiveness of mobile learning, followed by mobile learning system design. Wu et al. found 
that surveys and experiments were the primary research methods during that period. Also, their 
analysis showed that mobile learning in the context of higher education is most frequently 
researched (51.98%), followed by elementary school (17.51%), adult learning (12.43%), 
secondary/post-secondary education (8.47%), and disabled students’ learning (0.56%). With a 
more specific focus on mobile learning pedagogies, Hsu and Ching (2013) reported their review 
of nine quasi-experimental or experimental studies published between 2004 and 2011 on mobile 
computer-supported collaborative learning (mCSCL). They analyzed the studies regarding the 
following aspects related to mCSCL: student (education) level, content domain, group size, 
mobile software/applications that support collaborative learning, and mobile operating systems. 
They also reported the types of mCSCL interventions as well as their impact on students’ 
learning performance, engagement, participation, and interaction. 
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Although the aforementioned review articles can provide valuable input regarding what has 
been done, they might not be able to capture the latest and upcoming research agendas based on 
the consensus of the leading mobile learning researchers. The purpose of this paper is to research 
the collective insight from mobile learning research experts. This paper can help pave the way 
for meaningful and critical research in mobile learning. It also in part echoes the discussions 
regarding an agenda for mobile learning in a recent editorial by Rushby (2012). The authors aim 
to provide instances and directions that can help reveal the various facets and enrich the 
definitions of mobile learning. 


Method 

In order to inform the community of the latest research priorities in mobile learning, the 
Delphi method was selected as an appropriate method for systematically studying priorities in 
mobile learning research. The Delphi method is a consensus technique (Anderson & Kanuka, 
2003) that was developed in the 1950s by the RAND (Research and Development) Corporation 
to forecast the impact of technology on warfare. Infonnation on the RAND website defines the 
essential characteristics and goals of the Delphi method in the following statement: 

The method entails a group of experts who anonymously reply to questionnaires and 
subsequently receive feedback in the form of a statistical representation of the “group 
response,” after which the process repeats itself. The goal is to reduce the range of 
responses and arrive at something closer to expert consensus (RAND, n.d., para. 1). 

This method has been used previously to address various research questions, such as 
identifying research priorities in educational technology (Pollard & Pollard, 2004), K-12 online 
education (Rice, 2009), and video-sharing (YouTube) research (Snelson, Rice, & Wyzard, 2012). 

The current study invited experts in the field of mobile learning to identify priorities in 
mobile learning research. The expert panel participated in a three-round Delphi process 
involving two cycles of online questionnaires and feedback reports. Procedural steps for the three 
rounds of the Delphi study were as follows: 

Round 1 

Participants responded to the question, “What should be the research priorities for the field 
of mobile learning over the next 5 years?” The three authors of this paper reviewed the expert 
statements submitted in response to this question. Of these three, one is an expert in mobile 
learning technologies, another is experienced with the Delphi method, and the other has expertise 
in a variety of learning technologies and research methodologies. Each author reviewed and 
categorized the statements independently. 

Next, all three authors met to discuss the initial categories until consensus was achieved in 
both (1) identification and naming of each category and (2) placement of statements within each 
category. This process is consistent with the approach used in previous Delphi studies (e.g., 
Pollard & Pollard, 2004; Rice, 2009; Snelson et al., 2012). In all, the group of 14 experts 
generated 70 statements. These responses were consolidated into 40 statements and organized 
into 10 research priority categories. The statements and categories were used to develop the 
Round 2 survey. 
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Round 2 

Participants were asked to rate the consolidated statements on a Likert-type scale as to the 
degree of importance (1 = No Importance, 5 = High Importance). Additionally, experts were 
asked to prioritize the ten research categories (1 = Highest Priority, 10 = Lowest Priority). Once 
returned, descriptive statistics for the group ratings and rankings were calculated. 

Round 3 

Descriptive statistical information about how the group responded, as a whole, was 
provided to the participating experts. They were asked to review each statement and category, 
consider the group response and then re-rate the statements and re-rank the categories after 
taking the infonnation into account. The three-round Delphi process enabled the participants to 
generate their own opinions about necessary research areas, to prioritize research focus 
categories, and then to finalize their views based upon consideration from the entire group’s 
opinions. This process, engendering the dynamics of effective group interactions, enabled 
researchers to gain a consensus from a panel of expert participants in diverse geographical 
locations about mobile learning research priorities for the next 5 years. 

Delphi Panel Experts 

The recommended number of Delphi participants to include on an expert panel has varied, 
such as 10 to 30 (Anderson & Kanuka, 2003) and from 10 to 15 (Delbecq, Van de Ven, & 
Gustafson, 1975). For the current study, we identified and invited 59 experts to participate, and 
14 of them completed all three rounds of survey. The initial process of identifying experts was 
conducted through a comprehensive review of the literature similar to what was done in a 
previous Delphi study (i.e., Snelson et al., 2012). The process of identifying experts began with a 
keyword search in the titles of refereed journal articles through WorldCat, the World’s largest 
online catalog built and maintained by 72000 participating libraries in 170 countries and 
territories (Wikipedia, 2013). The keywords included tidiandhcld or ti:mobile learning or 
tiunlearning or tkm-leaming. The authors of mobile learning research were then categorized by 
the total number of articles they published and served as single/first author. Each author’s work 
was then further examined to make sure the invitation was sent to the authors with research 
relevant to mobile learning. For example, authors of research focusing on algorithm, mobile 
network establishment, mobile robots, or specific hardware on mobile devices (e.g., sensors) 
were excluded. After this process, a total of 41 experts who are first authors of mobile learning 
refereed articles were identified. Furthermore, four other sources were used to locate and identify 
18 additional experts to participate in this study. These included guest editors on mobile learning 
special issues in important journals in educational technology, editors of notable books on 
mobile learning (i.e., cited more than 10 times according to Google Scholar), and experts who 
were identified from the review of references of mobile learning refereed articles. In addition, 
one of the invited experts who could not participate recommended a list of mobile learning 
experts from Australia and New Zealand. An invitation was sent to 59 experts in mobile learning 
to participate in this Delphi study. Fourteen experts, all with doctoral degrees, completed three 
rounds of survey. Table 1 presents the demographic characteristics of the 14 Delphi panel 
experts. 
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Table 1: 


Mobile Learning Scholars Demographic Overview (Round 3 Participants, n = 14) 


Demographic Item 

n 

% 

Gender 



Male 

9 

64 

Female 

5 

36 

Highest Degree 



Doctorate 

14 

100 

Current Position Held 



University Professor 

12 

86 

University Researcher 

2 

14 

Discipline Area 



Archaeology 

1 

7 

Education 

8 

57 

Science & Engineering 

1 

7 

Information Systems/Technology 

2 

14 

Information Management 

1 

7 

Nursing 

1 

7 

Geographical Location 



(based on institution) 



Australia 

1 

7 

Canada 

1 

7 

Finland 

1 

7 

New Zealand 

1 

7 

Taiwan 

4 

29 

UK 

2 

14 

USA 

4 

29 


Note: Percentages are rounded and may not total 100% for some groups of demographic items. 


Findings 

In this section, we reported and discussed each of the 10 research categories and the 40 
research statements generated by the panel of experts to help inform the directions for future 
research in mobile learning. Among the 10 research categories, the need for research on learning 
and teaching strategies was identified as the top research priority category by this group of 
experts (median of 2.0). Following in priority is the need for research on affordances (of mobile 
technology) (median of 3.0). The complete frequency and percentages of expert rankings are 
shown in Table 2 with research priority categories arranged in order of median value. In 
addition, for each research category, we reviewed, selected, and summarized conceptual papers 
or empirical studies that are representative of the category. In doing so, we hope to provide 
examples of mobile learning research under each category identified by our panel experts. The 
inclusion of these studies does not indicate the categories have been saturated with mobile 
learning research. Instead, it shows what has been done, and how future research can build on the 
previous work in the areas considered important in the next five years by the panel experts. 
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Table 2: 

Research Priorities Ranked by Experts: Round 3 


Category 


Priority ranking, frequency 


Number of 1 
statements 


6 


8 


1. Learning and Teaching 
Strategies 

2. Affordances 

3. Theory 

4. Settings of Learning 

5. Evaluation/Assessment 

6. Learners 

7. Mobile Technologies 
and Interface Design 

8. Context Awareness and 
Augmented Reality 

9. Infrastructure and 
Management 

10. Country and Digital 
Divide 


3 

4 
6 
2 
4 
4 


0 


1 


1 


0 0 


0 0 


0 


1 


1 


1 


10 


0 0 0 0 


Note: A ranking of 1 equals ‘Highest Priority’ and 10 equals ‘Lowest Priority’. Mdn, medium value; 
IQR, interquartile range. 


Mdn 

(IQR) 

2.0 ( 2 . 0 ) 

3.0 (4.8) 
4.0 (2.0) 

4.5 (4.0) 
5.0 (4.5) 
5.0 (2.5) 
6.0 ( 1 . 8 ) 

7.5 (4.3) 
9.0 (2.0) 
10 . 0 ( 1 . 8 ) 


Research Priority Category 1: Learning And Teaching Strategies (6 Statements) 

The Delphi experts indicated the highest priority category is to examine strategies for 
applying mobile technologies to support learning and teaching. Delphi experts recommended six 
major strategies, including collaborative learning, game-based learning, inquiry-based learning, 
simulation, information-rich content delivery, and tutoring for context-aware ubiquitous learning. 
An example of research in this category, more specifically related to collaborative learning, is 
Boticki, Looi, and Wong (2011). In their study, Boticki et al. applied an experimental research 
design to study the use of mobile devices to facilitate communication and interaction among 
primary school students (aged 8 to 9) through a scaffold flexible grouping approach (first in 
pairs, then groups of 3 and 4). In their intervention, students could use their mobile devices to 
send invitations to peers who they identified as qualified group members. These members would 
have the necessary information to help complete the students’ task of finding complementary 
fractions to get the sum of 1. Table 3 lists the six research statements and descriptive statistics for 
the category of the highest research priority: learning and teaching strategies. In addition to 
facilitating communication and interaction, other types of intervention of using mobile 
technologies to support collaborative learning include presenting the individual portions of an 
assigned learning task and serving as the focal point of interaction (e.g., Roschelle et al., 2009), 
providing feedback for group learning and instructor teaching (e.g., Zurita and Nussbaum, 
2004b), and managing and regulating interaction process (e.g., Chen et al., 2008). 
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Table 3: 


Learning and Teaching Strategies 


Expert-generated statements 

Ratings of importance, frequency 


Research in: 

1 2 

3 

4 

5 

Mdn (IQR) 

Collaborative learning and cooperative 
learning 

0 0 

4 

5 

5 

4.0 (1.8) 

Game-based learning 

0 0 

2 

8 

4 

4.0 (0.8) 

Inquiry-based learning 

0 0 

1 

6 

7 

4.5 (1.0) 

Simulation teaching and learning 
strategies 

0 0 

1 

10 

3 

4.0 (0.0) 

How to develop information-rich 
content for delivery on mobile 
technology 

0 0 

2 

11 

1 

4.0 (0.0) 

Tutoring strategies for context-aware 
ubiquitous learning 

0 1 

5 

5 

3 

4.0 (1.0) 


Note: A rating of 1 equals ‘No Importance’ and 5 equals ‘Very High Importance’. 
IQR, interquartile range. 


Research Priority Category 2: Affordances (3 Statements) 

The next highest priority category for research in mobile learning involved investigating 
the affordances of mobile technologies to support teaching and learning. In the context of this 
paper, affordances refer to the learning or teaching tasks/activities made possible by the features 
or functions of mobile technologies (Hsu et al., 2014). The tasks and activities need to be guided 
by the strategies identified in Category 1: Learning and Teaching Strategies. The three research 
statements listed in Table 4 include the affordances of mobile technology to support: 1) problem 
representation and problem solving; 2) knowledge construction and collaborative learning; 3) 
contextualized learning. It is worth noting that collaborative learning appeared in both Category 
1 and Category 2, which shows that the experts value the importance and potential of mobile 
technology to support this type of learning. An example of research in Category 2: Affordances 
is the study conducted by Lai, Yang, Chen, Ho, and Chan (2007). In the experimental study of 
Lai et al., they found that mobile technologies are effective in improving 5 th graders’ knowledge 
creation during experiential learning on a field trip exploring plants. The students with Personal 
Digital Assistant (PDA) outperformed students without PDA as they were supported to recognize 
and utilize the capabilities (such as photo-taking and sound recording) afforded by the mobile 
technologies. For example, students with PDAs could record plant infonnation quickly by taking 
photos, compare multiple photos to examine different parts of a plant, and save and magnify a 
photo to examine a specific part of a plant more closely at a later time. Lai et al. also cautioned 
about early introduction of mobile technologies in experiential learning as that could make 
learners rely on mobile technology affordances (e.g., photo-taking) rather than utilizing their 
own ability (e.g., observation in the context of experiential learning). 
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Table 4: 


Affordances 


Expert-generated statements 

Ratings of importance, frequency 


Research in: 

1 2 3 4 5 

Mdn (IQR) 


Affordances of mobile technology to 
support problem representation and 
problem solving 

0 

0 

3 

7 

4 

4.0 (0.8) 

Affordances of mobile technology to 
support knowledge construction and 
collaborative learning 

0 

1 

1 

7 

5 

4.0 (1.0) 

Identifying clear, contextualized 
learning needs that mobile learning 
technologies and services could 
address 

0 

0 

1 

4 

9 

5.0 (1.0) 


Note: A rating of 1 equals ‘No Importance’ and 5 equals ‘Very High Importance’. 
IQR, interquartile range. 


Research Priority Category 3: Theory (4 Statements) 

Despite the fast-paced growing literature in the field of mobile learning (Kukulska-Hulme 
& Traxler, 2007) and the availability of mobile learning models (e.g., Koole, 2009; Park, 2011), 
experts still seem to recognize the needs for theoretical framework to guide the design and 
research of mobile learning by factoring in culture, context, ubiquitous learning and/or 
constructivist learning. Koole’s (2009) Framework for the Rational Analysis of Mobile 
Education (FRAME) model builds on the intersection of three fundamental aspects—device, 
learner, and social that allow educators to consider the overlapping aspects when designing 
mobile learning. These intersected aspects include device usability (i.e., device + learner), social 
technology (i.e., device + social), and interaction learning (i.e., learner + social). Finally, mobile 
learning is at the confluence of the device, learner, and social aspects. The FRAME model 
provides an intuitive way for considering and designing mobile learning activities. 

Park (2011) adopted Transactional Distance theory to create a pedagogical framework for 
mobile learning. Park’s framework categorized mobile learning into four types after analyzing 
the literature: 1) high transactional distance socialized m-leaming, 2) high transactional distance 
individualized m-learning, 3) low transactional distance socialized m-learning, and 4) low 
transactional distance individualized m-leaming—all mediated by mobile devices. The goal of 
this framework is to help instructional designers of open and distance learning to factor in the 
extent of psychological separation between the learner and the instructor as well as the levels of 
social/individual activities when designing mobile learning in their contexts. 

There are also other frameworks or models on different aspects of mobile learning, such as 
the 3-level evaluation framework of mobile learning proposed by Vavoula and Sharpies (2009) 
(see Research category 5: Evaluation/assessment), the self-regulated learning model of mobile 
learning proposed by Sha, Looi, Chen, and Zhang (2011) (see Research priority category 6: 
Learners), and the design requirements framework for mobile learning environments by Parsons, 
Ryu, and Crashaw (2007) (see Research priority category 7: Mobile technologies and interface 
design). With the Theory category ranked high (No.3) by the experts, it appears that the expert 
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panelists do not consider current models as sufficient for designing mobile learning in various 
contexts. Table 5 lists the four recommendations generated by the experts. 

Table 5: 

Theory 


Expert-generated statements Ratings of importance, frequency 


Research in: 1 

2 

3 

4 

5 

Mdn 

(IQR) 

Pedagogical framework for the use of 0 

mobile device for constructivist learning 

1 

2 

10 

1 

4.0 (0.0) 

Theoretical/conceptual framework to guide 0 

the design of mobile learning 

0 

0 

9 

5 

4.0 (1.0) 

Developing sound, culturally and 0 

contextually sensitive theories of learning 
for a mobile world 

0 

2 

12 

0 

4.0 (0.0) 

New pedagogical theories for context-aware 1 
ubiquitous learning environments 

0 

1 

12 

0 

4.0 (0.0) 


Note: A rating of 1 equals ‘No Importance’ and 5 equals ‘Very High Importance’. 
IQR, interquartile range. 


Research Priority Category 4: Settings For Learning (6 Statements) 

Settings for learning here refer to a variety of aspects, including level of education (e.g., 
tertiary education), delivery mode and environment (e.g., online, blended, traditional/classroom), 
and format and structure of education (formal vs. informal). An example of research under this 
category is Jones, Scanlon, and Clough (2013). Jones et al. compared and contrasted two case 
studies regarding the dimensions of learner control, location of learning, and the different 
support mechanisms for inquiry learning in informal settings. In the second case study, adult 
learners used mobile devices to engage in geocaching to learn about landscape. Jones et al. also 
proposed a framework for considering the aforementioned dimensions when developing mobile 
learning for infonnal settings. As seen in Table 6, the Delphi experts suggested a total of six 
research needs addressing mobile learning in various settings. As Wu et al. (2012) suggested in 
their review of 164 articles on mobile learning, 51.98% of research published between 2003 and 
2010 focused on higher education, and yet application in tertiary education is still considered 
important in Category 4. However, future research might want to consider conducting research at 
other levels (e.g., secondary education) to help fill the gap in research and contribute to the field. 
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Table 6: 


Settings for Learning 


Expert-generated statements 

Ratings of importance, frequency 


Research in: 

1 

2 

3 

4 

5 

Mdn (IQR) 

Application in tertiary education 

0 

2 

6 

5 

1 

3.0 (1.0) 

Smartphone technologies in the classroom 

2 

0 

4 

6 

2 

4.0 (1.0) 

Embedding mobile learning into day-to- 
day teaching 

0 

1 

3 

6 

4 

4.0 (1.5) 

Implementing mobile learning in a blended 
delivery mode of education 

0 

0 

4 

7 

3 

4.0 (0.8) 

Variations in student outcomes between 
online and traditional education 

2 

1 

6 

2 

3 

3.0 (1.0) 

Addressing how to bridge the gap between 
informal socialized learning and formal 
institutionalized learning, and the role that 
mobile learning might take in this 

0 

2 

3 

4 

5 

4.0 (2.0) 


Note: A rating of 1 equals ‘No Importance’ and 5 equals ‘Very High Importance’. 
IQR, interquartile range. 


Research Priority Category 5: Evaluation/Assessment (2 Statements) 

The two statements included in the evaluation/assessment research category, shown in 
Table 7, recommended research focus on developing methods for evaluating mobile learning in 
general. Experts also recognized the capability and potential of mobile technologies that lead to 
the context-aware ubiquitous mobile learning, and suggested developing strategies to assess 
mobile learning along those aspects. While there is little research on the methods and strategies 
of assessing mobile learning, there have been efforts in developing framework for evaluating 
mobile learning. For example, Vavloula and Sharpies (2009) proposed a 3-level framework 
consisting of usability (micro level), learning experience (meso level), and integration within 
existing educational and organizational contexts (macro level). They proposed to use this 3-level 
framework to address the challenges of evaluating mobile learning. Those challenges include: 1) 
capturing and analyzing learning in context and across contexts; 2) measuring mobile learning 
processes and outcomes; 3) respecting learner/participant privacy; 4) assessing mobile 
technology utility and usability; 5) considering the wider organizational and socio-cultural 
context of learning; 6) and assessing in/formality. The proposed framework and challenges 
provide good foundations for future research in developing and testing methods and strategies 
for evaluating mobile learning. 

From a different perspective, mobile technologies can be used to support assessment 
process and assessment can be considered a type of learning and teaching strategy. Nikou & 
Anatasios (2013) discussed mobile-device-based assessment such as classroom response systems 
(e.g., Clickers or other voting apps on mobile devices), self-assessment, peer assessment, 
collaborative assessment, computerized adaptive testing on mobile devices, dynamic assessment, 
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context-aware and location-aware assessment. When (formative) assessment is integrated into 
the instruction as part of students’ learning process by checking learners’ understanding and 
providing feedback, mobile assessment can serve as a variation of mobile learning which also 
deserves research attention. 


Table 7: 

Evaluation/Assessment 


Expert-generated statements 

Ratings of importance, frequencv 


Research in: 

1 

2 

3 

4 

5 

Mdn (IQR) 

Developing robust and appropriate 
methods to evaluate mobile learning 

0 

0 

0 

5 

9 

5.0 (1.0) 

Assessment strategies for context- 
aware ubiquitous learning 

0 

0 

3 

7 

4 

4.0 (0.8) 


Note: A rating of 1 equals ‘No Importance’ and 5 equals ‘Very High Importance’. 
IQR, interquartile range. 


Research Priority Category 6: Learners (4 Statements) 

Within the Learners research category, there are four statements as listed in Table 8. The 
Delphi experts valued researching how one’s self-directed learning capability affects mobile 
learning, with ten experts rating the statement 4 and one rating it 5. The statement, 
“psychological factors related to context-aware ubiquitous learning” also has the majority of 
experts rating it 4 but it was not as valued as the statement above. There is little research 
examining how self-directed learning capability can affect mobile learning or how to use mobile 
technology to support self-directed learning (SDL). However, one study by Sha, Looi, Chen, and 
Zhang (2011) examined the conceptual and theoretical connections between mobile learning and 
self-regulated learning (SRL). SRL is very often used interchangeably with SDL, except that 
SDL is often used in discussing web-based learning without instructors (Bracey, 2010). Sha et al. 
proposed an analytic SRL model as a conceptual framework for understanding mobile learning. 
They argued that in mobile learning learners need to assume the responsibility of their own 
learning, more so than in other types of learning, due to ubiquity afforded by mobile 
technologies. This need for learner responsibility makes self-regulated learning perspectives 
especially meaningful and important. In their model, self-regulation serves as the agency of 
learning that is mediated by mobile devices, whereas mobile devices (technologies) serve as 
social, cognitive, and metacognitive tools that can provide social and pedagogical supports for 
learner autonomy in the mobile learning processes. 


Research priorities in mobile learning: An international Delphi study 


12 



CJLT/RCAT Vol. 40(2) 


Table 8: 
Learners 


Expert-generated statements Ratings of importance, frequency 


Research in: 1 

2 

3 

4 

5 

Mdn (IQR) 

Cognitive variations in the Millennial 1 
Generation 

5 

5 

3 

0 

3.0 (1.0) 

How self-directed learning capability 0 
affects mobile learning 

1 

2 

10 

1 

4.0 (0.0) 

Meeting the learning needs of the 1 

newer generations of learners with 
mobile learning since they are 
comfortable using mobile technology 

3 

6 

3 

1 

3.0 (1.5) 

Psychological factors related to 0 

context-aware ubiquitous learning 

0 

5 

8 

1 

4.0 (1.0) 


Note: A rating of 1 equals ‘No Importance’ and 5 equals ‘Very High Importance’. 
IQR, interquartile range. 


Research Priority Category 7: Mobile Technologies and Interface Design (4 Statements) 

In this category (See Table 9), the experts identified the need in studying theory-based 
mobile system design and interface design for navigation/ease of use. The experts also consider 
that it is important to research mobile learning which incorporates Web 2.0 applications that 
foster collaboration, sharing, and creation with theoretical underpinnings of distributed 
cognition, sociocultural theory, and situated cognition (Hsu et ah, 2014). Clinical decision 
support tools were also valued, but not as much as other research statements, perhaps due to the 
specificity. Parsons, Ryu, and Cranshaw (2007) proposed a design requirements framework for 
mobile learning environments. They utilized the elements in their model as criteria to examine 
four mobile learning projects, exemplifying how to use the model for analyzing and designing 
mobile learning environments and systems. Their model factored in four perspectives, including 
generic mobile environment issues (e.g., mobile interface design), learning contexts, learning 
experiences, and learning objectives. This model also incorporated the interaction dimension that 
looks into the different needs under individual learning and collective learning. The study by 
Parsons et al. provided a good example of research needed in this category and a potential 
framework for research on developing new mobile learning technologies and systems. 
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Table 9: 


Mobile Technologies and Interface Design 


Expert-generated statements 

Ratings of importance, frequency 


Research in: 

1 

2 

3 

4 

5 

Mdn (IQR) 

Web 2.0 mobile learning 

0 

0 

6 

8 

0 

4.0 (1.0) 

Clinical decision support tools in 
appropriate disciplines 

0 

2 

5 

5 

2 

3.5 (1.0) 

Developing new mobile learning 
technologies and systems based on 
sound theory and analysis of 
mobilized learners 

0 

1 

2 

9 

2 

4.0 (0.0) 

Effective interface for mobile 
learning so that all learners can 
successfully learn using mobile 
technology 

0 

0 

3 

7 

4 

4.0 (0.8) 


Note: A rating of 1 equals ‘No Importance’ and 5 equals ‘Very High Importance’. 
IQR, interquartile range. 


Research Priority Category 8: Context Awareness and Augmented Reality (6 Statements) 

Experts recommended research in context-aware ubiquitous learning in general as seen in 
the statements listed in Table 10. One expert, particularly, focused on a few more specific 
statements on mobile augmented reality learning experiences in science, for collaborative 
problem solving, engagement, and learning gains. It is worth noting that context-aware 
ubiquitous learning also appears in other priority categories (e.g., 1,3,5) with various emphases 
including teaching and learning strategies, theory, and assessment, arguably due to the mobile 
devices’ affordances of ubiquity. 

The study by Shih, Chu, Hwang, and Kinshuk (2011) provides an example of research on 
context-aware ubiquitous learning. Shih et al. investigated the attitudes of 5 th graders and their 
teachers about participating in a context-aware ubiquitous learning activity (supported by 
wireless, mobile, and context-aware technologies) regarding campus vegetation. They found 
students had significantly positive changes in attitudes toward mobile ubiquitous learning. They 
also found that class management became easier for the teacher even when students were 
learning outdoors, as the students focused on the learning content presented on the mobile 
devices. Teachers were also able to better monitor individual progress and provide individual 
advice without constant need of directing students’ attention to learning tasks while in the field. 

In terms of research in mobile augmented reality, Dunleavy, Dede, and Mitchell (2009) 
investigated and discussed the affordances and limitations of mobile augmented reality learning 
that situated learners in a physical environment with the support of real-time immersive 
participatory simulations on mobile devices. According to Dunleavy et ah, while students and 
their teachers considered the learning experiences that involved technology-mediated narrative 
and interactive situated collaborative problem solving highly engaging, some students felt 
overwhelmed by the complexity of the activities. Their study presented the potential of mobile 
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augmented reality for learning and the challenges to be addressed through further research and 
design. 


Table 10: 

Context Awareness and Augmented Reality 


Expert-generated statements 

Ratings of importance, frequency 


Research in: 

1 2 

3 

4 

5 

Mdn (IQR) 

Context-aware ubiquitous learning 

0 0 

2 

5 

7 

4.5 (1.0) 

Adaptive mobile learning 

0 0 

4 

8 

2 

4.0 (0.8) 

To what extent can mobile augmented 

0 0 

6 

7 

1 

4.0 (1.0) 

reality experiences replicate authentic 






practices in high-need science-based or 






inquiry-based fields? 






To what extent can mobile augmented 

0 0 

7 

4 

3 

3.5 (1.0) 

reality experiences replicate, guide, and 






scaffold collaborative problem solving in 






physical environments? 






What mobile augmented reality design 

0 0 

5 

7 

2 

4.0 (1.0) 


strategies and techniques foster an 
effective blend of narrative, content 
delivery, instruction, problem solving, and 
game mechanics as measured by 
engagement? 


Do students immersed in mobile 114 7 1 4.0 (1.0) 

augmented reality experiences have 

greater learning gains on measures of 

science content and scientific inquiry 

process skills than students who are 

immersed in a similar paper-based control 

simulation? _ 

Note: A rating of 1 equals ‘No Importance’ and 5 equals ‘Very High Importance’. 

IQR, interquartile range. 


Research Priority Category 9: Infrastructure and Management (2 Statements) 

This priority category identified by the Delphi experts involved the “enabler” of mobile 
learning, including infrastructure and management. As seen in Table 11, the experts saw a need 
to examine the necessary facilities and physical settings (i.e., infrastructure) for mobile learning 
to occur and be sustained. There is little academic research in this category, probably due to its 
practical nature, leaning more toward strategic planning and implementation at the institutional 
level. However, the importance of this category should not be ignored. Cobcroft, Towers, Smith, 
and Axel (2006) identified opportunities and challenges brought forth by mobile learning 
through a literature review. They pointed out that institutions need to carefully consider wireless 
infrastructure, the stability of overall learning management system (LMS), and the capability of 
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LMS for cross-platform delivery of mobile learning. As this category still lacks research, future 
exploration could help fill the void in the literature and contribute to our understanding of 
various aspects of infrastructure and management that could enable mobile learning. 

Table 11: 


Infrastructure and Management 


Expert-generated statements 

Ratings of importance, frequency 


Research in: 

1 

2 

3 

4 

5 

Mdn (IQR) 

Infrastructure required for implementing 
mobile learning 

0 

0 

4 

6 

4 

4.0 (1.5) 

Management of mobile learning projects 

0 

1 

4 

5 

4 

4.0 (1.8) 


Note: A rating of 1 equals ‘No Importance’ and 5 equals ‘Very High Importance’. 
IQR, interquartile range. 


Research Priority Category 10: Country and Digital Divide (3 Statements) 

The last priority category identified by the Delphi experts involved researching the 
potential digital divide regarding mobile learning from different aspects (see Table 12). It 
addressed how mobile learning can benefit developing countries (i.e., mobile learning as 
solutions for education), how mobile learning is implemented in the most populated country (i.e., 
status report and case studies), and how to reduce digital divide in deploying mobile learning 
(i.e., solving digital divide to make mobile learning possible). Mobile learning has been 
primarily developed in Europe, the U.S., and some eastern Asian countries where there are 
continuous, and significant investment and rapid advancement in technology (Traxler & 
Kukulska-Hulme, 2005). Due to different (e.g., lower) levels of investment and advancement, 
mobile technologies and mobile learning can look very differently in practice and require 
appropriate adaptions in design for contextual needs. For example, Traxler and Dearden (2005) 
examined and evaluated the efficiencies, cost, and alternatives associated with using Short 
Message Service (SMS) to deliver learning content and study-guide materials with a goal of 
training 200,000 in-service teachers in Kenya (Sub-Saharan Africa). Traxler et al. found SMS 
has great potential for the Kenyan school system considering Kenya’s infrastructure, such as 
poor landline phone network, lively and energetic mobile phone networks, high mobile phone 
ownership and little or no Internet bandwidth outside of major cities. 

Table 12: 


Country and Digital Divide 


Expert-generated statements 

Ratings of importance, frequency 


Research in: 

1 

2 

3 

4 

5 

Mdn (IQR) 

How mobile learning can benefit 
developing countries 

0 

1 

4 

3 

6 

4.0 (2.0) 

Mobile learning in China's basic education 

2 

4 

6 

0 

2 

3.0 (1.0) 

Endeavoring to reduce digital divides in 
the deployment of mobile learning 
solutions 

0 

1 

7 

3 

3 

3.0 (1.0) 


Note: A rating of 1 equals ‘No Importance’ and 5 equals ‘Very High Importance’. 
IQR, interquartile range. 
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Discussions and Conclusions 

When comparing our research findings and the research foci identified in the content 
analysis study of Hwang and Tsai (2011), it appears that motivation and perceptions and 
attitudes of mobile learning that were highly researched between 2001 and 2010, were not 
considered as part of the Top 10 categories by our panel experts. However, ubiquitous learning is 
still considered important as it appears in a few categories ranked important by our panel of 
experts, Teaching and Learning Strategies (Category 1), Theory (Category 3), 
Evaluation/Assessment (Category 5), Learners (Category 6), Context Awareness and Augmented 
Reality (Category 8). The study using text-mining method by Hung and Zhang (2012) and the 
study using content analysis method by Wu at al. (2012) both found that past research focused on 
effectiveness of mobile learning and the design of mobile learning systems. The effectiveness of 
mobile learning can be researched in various directions and contexts. Considering the research 
category identified in this current study, the effectiveness of mobile learning can mean 
researching under categories such as Teaching and Learning Strategies (Category 1), 

Affordances (Category 2), Settings for Learning (Category 4), and Context Awareness and 
Augmented Reality (Category 8). Also, the design of mobile learning systems is still considered 
important by our panel of experts as that research focus aligns well with categories such as 
Mobile Technologies and Interface design (Category 7), and Context Awareness and Augmented 
Reality (Category 8). As our discussions of the alignment between past research and the findings 
of this current study have revealed, there are still many unchartered areas and unanswered 
questions in mobile learning research, such as Infrastructure and Management (Category 9) and 
Country and Digital Divide (Category 10). 

With the advancement in mobile technologies, proliferation of mobile devices, and mobile 
applications, mobile learning has garnered noticeable interest among educators and learning 
material developers. Also, research in mobile learning has gained great momentum in recent 
years. As it is an unfruitful task to identify isolated points of view that may lead to partial 
opinions, this Delphi study was designed to solicit the collective insight from scholars who make 
up some of the most prolific authors in mobile learning research. Participants responded to the 
question, “What should be the research priorities for the field of mobile learning over the next 5 
years?” The collection of 40 consolidated statements and ten priority categories, produced and 
ranked by the panel of participating experts, provide an answer to this question. The contribution 
of this study is in defining directions for future research in mobile learning. Each of the 40 
statements produced by the panel of experts offers a starting point that researchers may use to 
plan new studies or establish research agenda in mobile learning. The ten research priority 
categories, ranked by importance, align with existing discourse from researchers who have 
already turned their attention to various aspects of mobile learning research. Due to the practical 
value and potential of mobile technologies, the top-ranked priorities are related to the “how-to” 
of mobile learning (i.e., strategies) and “what it allows learners to do” (i.e., affordances). In 
addition, theory is emphasized (ranked 3 ld in research categories) for its importance in supporting 
the design of mobile learning and teaching strategies when considering pedagogies and 
technology affordances. 

The authors identify with the definition of mobile learning by O’Malley et al. (2005), and 
believe that mobile learning research should focus on learners and how to help them learn, and at 
the same time acknowledge the importance of affordances (i.e., what the technologies allow 
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learners to achieve) that interface between learners and learning technologies. Our research 
findings help reinforce this line of thinking, enrich the definition of mobile learning through the 
collective experts’ ideas, and provide prioritized categories, statements, and concrete examples 
for future research efforts. 


Limitations and Future Research 

While our study involved a representative international panel of experts, one limitation of 
our study is that not all of the 59 experts we identified could participate in our research. Also, 
some arguably important areas such as privacy and ethics in mobile learning practice have not 
made the Top 10 research categories. To amend the limitations, we provided the demographics 
of the expert panel so that readers can contextualize the findings of this study when reviewing 
this paper. Also, in the Acknowledgement section below, we listed the experts who are willing to 
be recognized to provide credibility and contextualize their consensus. In addition to pursuing 
the research directions identified by the panel experts in this study, future research could 
examine the most recent studies on mobile learning, and evaluate if the most recent studies and 
the suggested future research directions in those studies fall into the categories identified in this 
current Delphi study. 
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